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This study examined the North Carolina Virtual Public School’s (NCVPS) credit recovery 
program and other common credit recovery options available to students in the state. The 
study compared short- and longer-term academic outcome data across credit recovery 
options, as well as correlations between the academic outcomes and characteristics of 
enrolled students. Overall, NCVPS students were less likely than other credit recovery 
students to be economically disadvantaged, and a greater proportion of NCVPS students 
entered high school proficient in math and reading. However, there was little difference 
in short-term success rates (Such as end-of-course exam scores) between NCVPS credit 
recovery students and other credit recovery students. NCVPS students were less likely 
than other credit recovery students to graduate, but those who did graduate were more 
likely to graduate in four years. Among NCVPS students, outcomes for racial/ethnic 
minority students and economically disadvantaged students differed from those for 
other students. 


This brief summarizes the findings of Stallings, D. T., Weiss, S. P., Maser, R. H., Stanhope, D., 
Starcke, M., & Li, D. (2016). Academic outcomes for North Carolina Virtual Public School credit recov- 
ery students (REL 2017-177), Washington, DC: U.S. Department of Education, Institute of Edu- 
cation Sciences, National Center for Education Evaluation and Regional Assistance, Regional 
Educational Laboratory Southeast. That report is available at http://ies.ed.gov/ncee/edlabs/projects/ 
project.asp?projectID=4462. 
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Why this study? 


There is growing interest in North Carolina—and across much of the country—in strengthening the pres- 
ence of online learning in all public schools. In 2013 the North Carolina General Assembly directed the 
state to transition to digital learning tools that support anywhere-anytime, personalized, and student-cen- 
tered learning by 2017 (North Carolina General Assembly, 2013). The current study explores two facets of 
that transition: the extent to which online learning already plays a role in providing new options for stu- 
dents at risk of dropping out, and the range and scope of unaddressed or underaddressed needs of students 
in these online credit recovery programs. In addition to expanding understanding of participation in and 
outcomes related to online credit recovery, this study also demonstrates the range of online credit recovery 
data available for future analyses and how the data can be used. 


Viable credit recovery options are of great interest to the district-level personnel who are directly respon- 
sible for deciding which academic options to make available to students. Though North Carolina dropout 
rates have fallen and graduation rates have risen in recent years (North Carolina Department of Public 
Instruction, 2015a, 2015b), considerable public pressure remains for school districts to continue to improve 
these rates. In addition to the state-provided online credit recovery option offered through the North 
Carolina Virtual Public School (NCVPS), school districts use traditional options such as school-year and 
summer school course repetition in a face-to-face classroom setting, as well as third-party online credit 
recovery programs, to improve graduation rates. This report addresses the need for a better understanding 
of the similarities and differences across these programs. 


What the study examined 


This study addressed four primary research questions using data for 2008/09-2011/12: 


1. What are the features of and enrollments in North Carolina Virtual Public School (NCVPS) credit 
recovery courses and the other credit recovery programs available to students in North Carolina, and 
what are the characteristics of students who enroll in NCVPS and other credit recovery options? 


2. How do short-term success rates (as measured by scores on end-of-course exam retests) for students who 
complete NCVPS credit recovery courses compare with success rates for students who repeat a course 
through other credit recovery options? 


3. After completing an NCVPS credit recovery course, how do mid- to longer-term success rates (such as 
staying on track and graduating) compare with success rates for students who repeat a course through 
any other credit recovery option? 


As an extension to research questions 2 and 3, the study also analyzed outcomes for student subgroups 
within the NCVPS credit recovery sample: 


4. To what extent, if any, do success rates in subsequent exams, courses, dropout rates, and graduation 
rates for NCVPS credit recovery students vary by student subgroup? 


See Stallings et al. (2016) for descriptions of the major distinguishing features of the credit recovery options 
and the appendix of the current report for the data and methods used to answer these questions. 


What the study found 


The study first focused on information about online and face-to-face credit recovery options in North Caro- 
lina, as well as comparisons of student characteristics by credit recovery option. The study then used simple 
comparisons to note differences between various short- and longer-term academic outcomes, followed by 
statistical analyses to identify possible relationships between those outcomes and various characteristics of 
credit recovery students. 


Traditional, school-year course repetition remains the most common credit recovery option in North Carolina, but 
online and summer school credit recovery enrollments are growing fast 


Courses retaken in a traditional, face-to-face classroom setting during the school year are the most common 
credit recovery options available to North Carolina public school students. How these options are imple- 
mented (such as which courses are offered, how long students must participate to earn credit, and per 
student cost) varies across school districts and school years. 


While traditional, school-year course repetition continues to be the most frequently used credit recovery 
option in North Carolina, both traditional summer school and NCVPS credit recovery options have made 
enrollment gains in recent years (figure 1).! 


Overall, traditional, school-year credit recovery courses are by far the most common credit recovery 
option, but enrollment in these courses declined from 2008/09 to 2011/12 at every grade level; the declines 
ranged from 7 percent in grade 12 to 25 percent in grade 9. Summer school credit recovery courses are 


Figure 1. Overall, traditional, school-year credit recovery courses are by far the most common 
credit recovery option, 2008/09-2011/12 
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tion of face-to-face courses. 
Source: Authors’ analysis of transcript data provided by the North Carolina Virtual Public School and the Education Research Data 


Center. 


more common than NCVPS credit recovery courses. In 2011/12 summer school courses accounted for 
16-18 percent of total credit recovery courses in grades 9-11 and 4 percent in grade 12, while NCVPS 
credit recovery courses accounted for 4-6 percent of total credit recovery courses across all grades. From 
2008/09 to 2011/12 enrollment increased in both summer school (52 percent) and NCVPS credit recovery 
courses (67 percent). 


The same general pattern holds by individual course. Traditional school-year courses are the most common, 
summer school courses are the second most common, and NCVPS courses are the least common (figures 
2 and 3). However, enrollment in all NCVPS credit recovery courses grew from 2008/09 to 2010/11. It 
declined slightly from 2010/11 to 2011/12 but still accounted for between 2.2 percent (World History) and 
6.4 percent (Algebra II) of enrollments. 


North Carolina Virtual Public School students were less likely than their peers in traditional credit recovery programs 
to be economically disadvantaged, and a greater proportion entered high school proficient in math and reading 


While NCVPS credit recovery students were similar to other credit recovery students in many ways, there 
were some notable differences across student subgroups. 


Race/ethnicity. Enrollment in NCVPS was similar to enrollment in traditional school-year credit recov- 
ery courses across all racial/ethnic groups. However, Black students consistently accounted for a larger 


Figure 2. For individual credit recovery courses with an end-of-course exam, traditional, school- 
year courses are the most common, summer school courses are the second most common, and 
North Carolina Virtual Public School courses are the least common, 2008/09-2011/12 
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Source: Authors’ analysis of transcript data provided by the North Carolina Virtual Public School and the Education Research Data 
Center. 


Figure 3. For individual credit recovery courses without an end-of-course exam, traditional, school- 
year courses are the most common, summer school courses are the second most common, and 
North Carolina Virtual Public School courses are the least common, 2008/09-2011/12 
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Note: Because of missing data or errors in the administrative data related to students’ grade level, enrollment totals in figures 1, 
2, and 3 do not match. Traditional, school-year course repetition and traditional, summer school course repetition refer to repeti- 
tion of face-to-face courses. 


Source: Authors’ analysis of transcript data provided by the North Carolina Virtual Public School and the Education Research Data 
Center. 


proportion of enrollment in summer school courses than in either of the other credit recovery options 
(41-50 percent), and White students consistently accounted for a smaller proportion of enrollment in 
summer school courses (34—40 percent) than in the other options (table 1). 


Economically disadvantaged status. Students who were economically disadvantaged (as proxied by eli- 
gibility for the federal school lunch program) accounted for a larger proportion of enrollment in summer 
school credit recovery courses than in either of the other credit recovery options (table 2). 


Students receiving special education services. The proportion of students receiving special education 
services was similar across credit recovery options and fairly stable over time. Between 2008/09 and 2011/12 
students receiving special education services accounted for 14-18 percent of enrollment in traditional 
school-year and NCVPS credit recovery courses and 16-17 percent of enrollment in summer school courses. 


Academic characteristics of enrolled students. In most years, for both math and reading, the percentage 
of students reaching proficiency on the grade 8 end-of-grade exams (the last state exam taken before enter 
ing high school) was higher among students in NCVPS credit recovery courses than among students in 
other credit recovery options (table 3). 


Table 1. Race/ethnicity of credit recovery students in North Carolina, by credit recovery option, 
2008/09-2011/12 (percent) 


Credit recovery option 


and race/ethnicity PXINtY:¥A0):) 2009/10 2010/11 2011/12 
North Carolina Virtual Public School credit recovery 

Black 45.3 44.1 42.3 41.6 
Hispanic 6.0 8.6 11.2 13.7 
Other 4.2 4.3 4.5 6.1 
White 44.5 43.1 40.1 38.1 
Traditional, school-year course repetition 

Black 40.6 43.0 43.6 42.0 
Hispanic 9.8 10.9 12.7 13.9 
Other 5.5 5.4 6.2 6.4 
White 44.0 40.7 37.6 37.7 
Traditional, summer school course repetition 

Black 48.3 49.7 50.1 48.3 
Hispanic 7.5 7.3 10.3 12.3 
Other 4.2 5:1 5.4 5.5 
White 39.8 38.0 34.2 33.9 


Note: Black includes African American. Hispanic includes Latino. Other includes Asian, American Indian/Alaskan Native, Hawaiian 
Native/Pacific Islander, and multiracial. Traditional, school-year course repetition and traditional, summer school course repetition 
refer to repetition of face-to-face courses. 


Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 


Table 2. Percentage of credit recovery students in North Carolina who are economically 
disadvantaged, by credit recovery option, 2008/09-2011/12 (percent) 


Credit recovery option 2008/09 2009/10 2010/11 2011/12 
North Carolina Virtual Public School credit recovery 47.6 51.9 52.1 55.9 
Traditional, school-year course repetition 47.3 56.7 64.0 67.5 
Traditional, summer school course repetition 50.0 57.0 60.3 63.9 


Note: Traditional, school-year course repetition and traditional, summer school course repetition refer to repetition of face-to-face 
courses. 


Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 


Table 3. Percentage of credit recovery students reaching proficiency on the grade 8 end-of-grade 
exam in North Carolina, by content area and credit recovery option, 2008/09-2011/12 


Content area and credit recovery option 2008/09 2009/10 2010/11 2011/12 
Math 

North Carolina Virtual Public School credit recovery 48.0 53.3 58.0 64.1 
Traditional, school-year course repetition 53.2 47.8 49.0 57.7 
Traditional, summer school course repetition 45.2 48.5 57.9 63.7 
Reading 

North Carolina Virtual Public School credit recovery 77.6 66.6 52.7 49.2 
Traditional, school-year course repetition 80.2 70.5 47.6 41.0 
Traditional, summer school course repetition 69.9 55.5 43.3 45.1 


Note: Traditional, school-year course repetition and traditional, summer school course repetition refer to repetition of face-to-face 
courses. 


Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 
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North Carolina Virtual Public School students scored lower on state end-of-course exam retests after credit recovery 
than did traditional credit recovery students, but the differences were small once differences between student 
subgroups were taken into account 


The study compared two short-term academic outcomes for NCVPS credit recovery students—scaled 
scores and achievement levels on standardized North Carolina end-of-course exams retaken after complet- 
ing credit recovery—to outcomes for all other credit recovery students (traditional face-to-face students, 
summer school students, and other identifiable credit recovery students, such as those who participated 
in third-party credit recovery options not provided by the state). At the individual course level and across 
all courses, the NCVPS credit recovery students in the sample years reached proficiency on end-of-course 
exam retests at lower rates (0.3-19.6 percentage points) and scored lower on end-of-course exam retests 
after credit recovery (0.1-2.6 points) than did other credit recovery students (table 4). 


Regression analysis was applied to adjust for other possible influences that might be related to the dif 
ferences in retest scores (such as credit recovery option, student demographics, and prior scores on the 
same exam). NCVPS credit recovery students still appeared to be less likely to score as high as or to reach 
proficiency as often as other credit recovery students after credit recovery. The differences were statistically 
significant, but because the effect sizes related to the differences were very small (all below 0.005), the dif 
ferences were not particularly meaningful. 


In other words, while there was no indication that participation in NCVPS credit recovery was more ben- 
eficial in terms of shorter-term outcomes for those students than was participation in other credit recovery 


Table 4. Credit recovery students’ performance on North Carolina end-of-course exam retest after 
completing credit recovery, by course and credit recovery option, 2008/09-2011/12 
NCVPS credit All other credit 
Outcome and course recovery students recovery students? Difference 


Students reaching proficiency on end-of-course exam retest (percent) 


English | 34.6 34.9 -0.3 
Algebra | 29.5 36.1 -5.6 
Algebra II 45.5 55.6 -10.1 
Geometry 29.9 49.5 -19.6 
Physical Science 30.8 45.0 -14.2 
Biology 26.9 41.2 -14.3 
Civics & Economics 27.1 36.0 -8.9 
U.S. History 31.7 43.1 -11.4 
Average scaled score on end-of-course exam retest 

English | 142.8 142.9 -0.1 
Algebra | 143.3 144.7 -1.4 
Algebra II 145.1 146.9 -1.8 
Geometry 144.2 146.8 -2.6 
Physical Science 144.3 146.8 -2.5 
Biology 142.3 144.6 -2.3 
Civics & Economics 144.0 145.3 -1.3 
U.S. History 145.7 147.0 -1.3 


NCVPS is North Carolina Virtual Public School. 


a. Includes traditional face-to-face students, summer school students, and other identifiable credit recovery students, such as 
those who participated in third-party credit recovery options. 


Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 


options, there also was no definitive indication that it might have been notably detrimental (see the appen- 
dix for a brief explanation of statistical significance and effect sizes; see Stallings et al., 2016, for these and 
other regression results). 


Longer-term outcomes for North Carolina Virtual Public School credit recovery students relative to outcomes for other 
credit recovery students were mixed 


The study compared four longer-term education outcomes for NCVPS and other credit recovery options: 
e Student re-enrolled in school the following year. 
e Student reached proficiency in the subsequent course in the course series—an indicator derived by 
stringing together the sequence of students’ courses in their longitudinal records. 
e Student graduated from high school. 
e Student graduated from high school in four years (that is, on-time graduation). 


Re-enrollment and subsequent-course proficiency. Between 2008/09 and 2011/12 NCVPS credit recovery 
students re-enrolled in school the following year at a rate of 91.7 percent, 2.7 percentage points higher than 
other credit recovery students. NCVPS credit recovery students’ academic success rates in the next course 
in each sequence, relative to students who completed other credit recovery options, varied between 3.9 per- 
centage points lower and 2.4 percentage points higher (table 5). 


As with the short-term outcomes analysis, regression analysis was applied to determine whether the differ- 
ences would remain after other possible explanatory factors (such as differences in special education status 
and school attendance rates) were accounted for. NCVPS credit recovery students still appeared to be more 
likely than their peers to re-enroll in school the year following their credit recovery coursework, but the 
regression analysis eliminated the positive results for academic success in the next course in the English 
II-III and the English III-IV course sequence. 


Graduation. NCVPS credit recovery students graduated at a lower rate (76.0 percent) than did other credit 
recovery students (77.7 percent; table 6). Even after regression analyses were applied to control for several 
other explanatory variables, NCVPS credit recovery students were still statistically significantly less likely 
to graduate than other credit recovery students were. However, among credit recovery students who gradu- 
ated, NCVPS credit recovery students were more likely than other credit recovery students to graduate on 
time. 


Table 5. Longer-term in-school outcomes among credit recovery students in North Carolina after 
completing credit recovery, by credit recovery option, 2008/09-2011/12 (percent) 


NCVPS credit All other credit 

Outcome recovery students recovery students? Difference 
Student re-enrolls following year 91.7 89.0 +2.7 
Student passes next course in sequence 

English Il after English | credit recovery 67.1 68.0 -0.9 
English Ill after English Il credit recovery 73.6 71.2 +2.4 
English IV after English III credit recovery 88.4 86.9 +1.5 
Algebra Il or Geometry, after Algebra | credit recovery 60.8 64.7 -3.9 


NCVPS is North Carolina Virtual Public School. 


a. Includes traditional face-to-face students, summer school students, and other identifiable credit recovery students, such as 
those who participated in third-party credit recovery options. 


Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 


Table 6. Longer-term graduation outcomes among credit recovery students in North Carolina after 
completing credit recovery, by credit recovery option, 2008/09-2011/12 (percent) 


NCVPS credit All other credit 
Outcome recovery students recovery students? Difference 
Student graduates 76.0 77.7 -1.7 
If student graduates, graduates on time (4 years or less) 31.5 24.3 +7.2 


NCVPS is the North Carolina Virtual Public School. 


a. Includes traditional face-to-face students, summer school students, and other identifiable credit recovery students, such as 
those who participated in third-party credit recovery options. 


Source: Authors’ analysis of administrative data provided by the Education Research Data Center. 


Among North Carolina Virtual Public School credit recovery students, outcomes for racial/ethnic minorities and 
economically disadvantaged students differed from those for other students 


The study also examined academic outcomes within the population of NCVPS credit recovery students 
only. The goals were to note differences between outcomes for different student subgroups and identify 
possible correlations between these outcomes and key student characteristics. Findings from these analyses 
included: 

e Racial/ethnic minority NCVPS credit recovery students frequently had higher success rates than 
White NCVPS credit recovery students, not only in terms of re-enrolling the year following credit 
recovery but also in terms of graduating and graduating on time. 

e Economically disadvantaged NCVPS students had lower graduation rates after credit recovery 
than did NCVPS students who were not economically disadvantaged. 


These outcomes persisted even after regression analyses controlled for other possible explanatory variables 
(such as differences in special education status and school attendance rates). See Stallings et al. (2016) for 


more details on these regression results. 


Implications of the study findings 


While there were almost no clear and strong positive associations between participation in NCVPS credit 
recovery courses and the short and longer-term academic outcomes examined relative to participation in 
other credit recovery options, there also appeared to have been few meaningfully negative associations. 


As programs like NCVPS continue to grow (NCVPS is now the second-largest online school in the nation, 
after Florida’s; Hughes, Zhou, & Petscher, 2015) and states continue to transition to digital learning, it will 


become increasingly important to expand understanding of how to best support that transition. 


Limitations of the study 


The large number of cases examined reduces the chance of the analyses being affected by unusual results 
from a single year, but the size of the dataset does not offset several important limitations. 


First, because the data were not derived in a controlled experimental environment, the analyses cannot be 
used to make any direct conclusions about the quality of online credit recovery courses or of their overall 
efficacy in improving student academic outcomes. However, the analyses do provide the kind of informa- 
tion needed to understand how much and what kind of data are available on the topic, as well as to identify 
some of the causal research questions the available data may allow future studies to consider. 
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Second, the state funding formula for NCVPS credit recovery changed in the 2011/12 school year. The 
study’s analytical design was not affected by this funding change, as all credit recovery students in the 
sample were enrolled before the funding change. However, participation, and thus the correlations, might 
change under the new funding regime. 


Third, end-of-course exams in North Carolina do not cover all subjects; the tested subjects have changed 
over time, and the exams themselves have recently changed. Anticipating future results on the basis of past 


testing scenarios is not always practicable. 


Finally, the samples used to conduct the analyses are not necessarily representative of all online credit 
recovery programs or participants. 


10 


Appendix. Data and methodology 


This appendix includes an overview of the data used for this report, as well as of the methods used to com- 
plete the analyses. 


Data 


This section outlines the sources of the data used for this report and provides details about the student data 
sample. 


Sources. The data for this study are from three primary sources: North Carolina Virtual Public School 
(NCVPS), which provided a description of its credit recovery program and student enrollment data that 
could be matched to state administrative data; longitudinal, administrative student records provided by the 
North Carolina Department of Public Instruction and hosted and maintained by the Education Research 
Data Center at Duke University; and the seven third-party credit recovery program vendors in operation in 
the state, who provided descriptions of their programs. Administrative records were at the student level and 
included many individual demographic variables—for example, age, race/ethnicity, and economically dis- 
advantaged status (as proxied by eligibility for the federal school lunch program)—and academic variables 
—for example, performance on standardized state end-of-course exams and course enrollment. 


Sample. The sample included all identifiable students who took credit recovery courses offered by NCVPS 
between 2008/09—the first full academic year of NCVPS’s program—and 2011/12, as well as all identifiable 
students whose credit recovery involved re-enrolling in a course during the regular school year (traditional 
face-to-face credit recovery), during summer school, or, when possible, as part of some other credit recovery 
program, such as participation in a program provided by a third-party vendor. For some research questions, 
student records from the preceding year (2007/08) were included to provide baseline (pre-participation) 
measures for some students. 


Because students can fail more than one course and can therefore choose more than one type of credit 
recovery, for most analyses data were organized by instances of course enrollment instead of by student. 
Thus a single student could have separate records in the data for credit recovery courses taken online 
or face to face, as well as separate records for courses taken in different years. The sample of all credit 
recovery students included about 14,900 instances of NCVPS credit recovery enrollment between fall 2008 
and spring 2012, about 329,900 instances of traditional school-year credit recovery enrollment, and about 
33,900 instances of summer school credit recovery enrollment (see figures 1-3 in the main text). Courses 
included in the analyses were limited to the 12 offered by NCVPS as part of its credit recovery program 
—English I, English II, English II, English IV, Algebra I, Algebra II, Geometry, Physical Science, Biology, 
Civics and Economics, U.S. History, and World History. 


The sample used for analyses of differences among NCVPS credit recovery students included only the 
12,900 students for whom complete administrative data could be matched to evidence of participation in 
NCVPS credit recovery. 


In some cases a subset of students was excluded for some research questions, because the students had 
not been enrolled long enough to experience some of the mid- or longer-term outcomes of interest. In 
other cases students were excluded when their records could not be matched to demographic information 
required for analysis. However, rates of matching students to the demographic and transcript data needed 
for many of the analyses were very high (for example, 93 percent of NCVPS credit recovery students were 
identifiable in the official state administrative dataset). 
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Quantitative analysis methodology 


This section outlines the strategies used to determine categorization of each record of enrollment in a 
credit recovery course and provides details about the regression analyses conducted after completion of this 
categorization. 


Identifying NCVPS online credit recovery courses. Because the North Carolina Department of Public 
Instruction administrative database does not contain an online course identifier and because individual 
districts are responsible for encoding student enrollment, the process of identifying the various instances 
of credit recovery required several steps. The first step was to match NCVPS'’s records to the state records, 
resulting in a very high match rate (93 percent). 


Identifying traditional school-year and summer school credit recovery courses. Next, records of all other 
courses for the years of interest were selected, then reduced to include only those that matched the 12 
NCVPS credit recovery courses. Despite differences in the course titles and in course coding schemes 
across districts, all districts include a universal initial fournumber course code that identifies the discipline 
(first two numbers) and the course (second two numbers). By identifying instances of targeted courses that 
were repeated (and for which initial enrollment included an indication of no credit earned), the study team 
could deduce (from information about the semester in which the repeated courses were taken) whether the 
courses were traditional (that is, taken in the fall or spring semester) repetitions or summer school repeti- 
tions. Because credit recovery requires repeating courses and because the transcript data begin in 2007/08, 
the credit recovery course analyses are limited to 2008/09-2011/12. 


Overview of quantitative analyses. For all components of research questions 2 and 3, the study team used 
multivariate ordinary least squares regression models for continuous outcomes (for example, scaled test 
scores) and multivariate logistic regression models for binary outcomes (for example, whether a student 
graduated). All regression analyses compared two populations: students who took an NCVPS credit recov- 
ery course and credit recovery students who did not. 


Analyses of short-term outcomes. The population for these analyses included all students who retook a 
state standardized end-of-course exam after completion of credit recovery. The study team used a multi- 
variate ordinary least squares regression to predict students’ course-specific normalized end-of-course 
exam scores following NCVPS credit recovery participation and multivariate logistic regression to predict 
whether a student reached proficiency following credit recovery. 


Analyses of mid- and longer-term outcomes. The population for these analyses included all students for 
whom dropout data or re-enrollment data were available after taking a credit recovery course. Analyses 
related to success in subsequent courses were limited to the Algebra I~Algebra II/Geometry sequence and 
the English I-English II-English II]-English IV sequence. All analyses for the components of research 
question 3 employed multivariate logistic regressions. 


Statistical significance and effect sizes. When the results of a regression analysis with many potential 
outcomes are statistically significant (that is, not likely to have happened by chance), there is a temptation 
to assume that the results are also meaningful. What is often more important is the size of the predicted 
result. The effect size helps put into context how meaningful a statistically significant result is. This study 
considers effect sizes between 0.2 and 0.5 to be small, effect sizes between 0.5 and 0.8 to be medium, and 
effect sizes of 0.8 and above to be large (Cohen, 1988). 
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Note 

There is a fourth, smaller group of credit recovery students in North Carolina: those enrolled in 
third-party (that is, not state-provided) credit recovery programs. However, because limitations in 
the data prevent accurate identification of all those students, the third-party credit recovery students 
who were identifiable for the study period may not be demographically representative of all third-party 
online credit recovery students in those years. Therefore, the analyses of enrollment presented in this 
section and the section that follows do not include data for those students. Identifiable third-party 
credit recovery students are included with traditional school-year and summer school credit recovery 
students as part of the non-NCVPS credit recovery comparison group used in the analyses of academic 
outcomes described in the second half of this brief. 
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